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Overview 


•  Open  architectures  (OA)  and  reusable  software 
components  offer  the  promise  of  more  rapid  fielding 
of  increments  in  systems  development 


•  Testing  and  re-testing  these  components  requires  a 
significant  level  of  effort  as  new  systems  are 
developed  and  old  systems  are  upgraded 
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•  How  much  testing  is  enough?  When  can  we  stop 
Viiiij  testing? 
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Even  good  strategies 
may  reach  a  point  wher 
continued  testing  yields 
no  new  information 
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Motivation 
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Good  testing  strategies  offer  the 

Hinh 

most  information  per  unit  cost 

Knowledge  of 
or  confidence  in 
system  operation 
under  load 


Low 


Low 

Cost  of  testing  in 

High 

budget  and  schedule 
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Model  fundamentals 


•  We  can  identify  good  testing  strategies  by  constructing  a  simple 
model  of  the  system,  its  components,  and  its  attendant  test  suite 


This  model  requires 

-  Estimates  of  a  prior  probability  of  failure  for  modules  within  the  system 

-  Estimates  of  the  coverage  for  each  test  in  the  suite  over  these  modules 

These  estimates  need  not  be  precise  to  make  the  model  useful 

-  Monte  Carlo  simulation  can  be  used  to  sample  around  the  estimates  as 
means,  offering  some  insight  into  model  sensitivity 
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Model  fundamentals 
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•  This  model  should  help  answer  questions  like: 


-  Given  a  desired  level  of  confidence  in  system  operation,  how  much 
testing  should  we  accomplish?  How  much  will  this  cost? 

-  Given  a  fixed  budget  for  testing,  how  much  confidence  in  system 
performance  can  we  achieve  through  testing? 

-  Given  a  particular  test  suite,  how  much  information  is  attainable  given 
infinite  resources? 
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Model  fundamentals 
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A  module  is  modeled  as  a 
unit  circle  with  probability 
of  being  defective  bt 

Test  Tx  exercises  region  Aix 
in  module 

In  general  we  assume  that  Tx 
may  exercise  several  regions 
across  several  modules 


A  test  has  two  possible  outcomes: 

-  PASS  indicates  that  the  test  did  not  detect  a  defect  in  any  of  the 
exercised  regions  within  the  modules  tested 

-  FAIL  indicates  that  at  least  one  module  exercised  is  defective, 
though  we  may  not  know  which  one 
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Model  fundamentals 


•  These  ambiguities  offer  a  rich  framework  for  modeling  realistic 
system  testing  scenarios 

-  We  need  not  execute  (and  pay  for)  Tx  to  forecast  information  returned 

-  Within  this  language  of  expression  we  can  formulate  a  quantitative 
assessment  of  the  information  returned  by  a  test  sequence 


•  Across  the  system  of  modules  M;  we  can  measure  the 

information  returned  by  a  test  using  the  classic  residual  entropy 
for  a  distribution  of  probabilities: 

IILL 


H  =  Z h,  =  £ -b,  log,  bt  -  (1  ■ - b, )  log2  (1  -  b, ) 
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Model  fundamentals 


At  maximum  entropy  we  have  a  50/50  chance  that  our 
module  is  good  or  bad — we  might  as  well  flip  a  coin 
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Model  fundamentals 
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•  From  entropy,  we  derive  the  forecast  measure: 


Q(Tx )  =  (max(Z>/“/ ,  1  - b(ml )P(TX  fails)  +  max(/?/’1'55 , 1  - bfass  )P(Tx  passes)) 

i 

•  Let  cx  be  the  cost  of  executing  test  Tx  in  appropriate  units  of 
time  or  money  (or  both)  A  good  strategy  will  sequence  the 
suite  of  tests  such  that: 

Q(Tm)  ^  Q(Tm)  ^  Q(T[m]) 

I  C[l]  C[2]  C[m] 

P 

<1  •  These  ratios  represent  information  per  unit  cost 
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Model  implementation 
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Risk- based  Testing  System  Simulation 
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If ill1 


Execution  Parameters 
Case  Name 
Random  Seed 
Number  of  Trials 
Defects  per  Trial 


Reconfigure  tests  per  trial?  yes 

■ _ _ 

Strategy 


random 


System  Generation  Parameters 

Number  of  Modules 


Number  of  Tests 

Modules  per  Test 
Tests  per  Module 
Coverage  per  Test 
Failure  rate 


20 


Min 


Max 


0.100 


0.10000000 


default 
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Model  implementation 


A  r\  o 


Risk- based  Testing  System  Simulation 


If ill1 


Analysis  Parameters 


zerodefect 
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Current  Chart 


Information  measures 
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Model  implementation 


A  r\  o 


Risk- based  Testing  System  Simulation 
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Model  implementation 


A  r\  o 


Risk- based  Testing  System  Simulation 
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Model  implementation 


A  r\  o 


Risk- based  Testing  System  Simulation 
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Model  implementation 


A  r\  o 


Risk- based  Testing  System  Simulation 
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Model  implementation 
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Risk- based  Testing  System  Simulation 
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Model  implementation 
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Risk- based  Testing  System  Simulation 
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Model  implementation 
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Risk- based  Testing  System  Simulation 
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Model  implementation 
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Risk- based  Testing  System  Simulation 
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But,  what  does  it  all  mean? 


i 


•  Effective,  cost-efficient  testing  is  critical  to  the  long-term 
success  of  open  architecture  programs 

•  This  model  and  prototype  decision  aid  provide  a  rigorous  yet 
tractable  way  ahead  to  improve  system  testing 

•  Using  this  framework  we  can  build  the  tools  to: 

-  Lower  the  testing  costs  for  a  given  level  of  system  reliability 

-  Improve  the  use  of  existing  suites  for  a  given  budget  or  schedule 

-  Design  better,  more  targeted  test  suites  to  minimize  redundancy 

-  Provide  insight  into  the  power  or  sensitivity  of  current  test  suites 
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